Offload routed expert of RL - #1957

@claude review

Claude: ## Summary

Adds offload_rollout_routed_experts config flag to keep routed-expert tensors on CPU and move each layer's slice to GPU on-the-fly during forward. Also adds per-step max_memory / reserved_memory logging to the RL training loop.

ProduceBatchResult impact: not affected
RoutedExperts impact: ownership unchanged — tensors remain owned by the trainer worker; only device placement changes (CPU vs GPU when flag is True). No new retention paths introduced.

Issues

Warning

• [xtuner/v1/rl/trainer/worker.py:794-795] max_memory_allocated() / max_memory_reserved() are monotonically non-decreasing without a prior reset_peak_memory_stats() call. Unlike the SFT path (which resets per step), the RL path will log identical values after the first peak. See inline comment for details.

Nit

• Commit message "offload r3" doesn't follow the project convention [Tag] Short description (e.g. [Feature] Offload rollout routed experts to CPU).
• [xtuner/v1/module/decoder_layer/moe_decoder_layer.py:643-645] The CPU→GPU transfer is synchronous. pin_memory() + non_blocking=True would be the natural extension when prefetch is implemented.

Verdict

APPROVE — the core logic is correct and well-scoped. The offload flag cleanly integrates with the existing SequenceContext.to() and _pre_moe_forward paths. The memory metric issue is minor (monotonic values are still useful for cross-run comparison).

@claude approve

Claude: Approved as requested. The PR adds a clean offloading mechanism for rollout routed experts to CPU, reducing GPU memory pressure when large batches are dispatched to train workers. The implementation correctly gates the .to(device) call in SequenceContext.to() and performs per-layer on-demand transfers in _pre_moe_forward. Memory logging (max_memory, reserved_memory) was also added to WorkerTrainLogItem.